Skip to content

Conversation

@QiWang19
Copy link
Member

@QiWang19 QiWang19 commented Nov 13, 2025

- What I did

- How to verify it

  1. launch cluster with this patch
4.20.0-0.nightly-2025-11-22-055654, openshift/machine-config-operator#5414 (gcp) 
  1. apply a ClusterImagePolicy name: openshift
oc create -f clusterimgpolicycr.yaml

# clusterimgpolicycr.yaml

apiVersion: config.openshift.io/v1
kind: ClusterImagePolicy
metadata:
  name: openshift
spec:
  scopes:
  - "example.com/test"
  policy:
    rootOfTrust:
      policyType: PublicKey
      publicKey:
        keyData: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFVW9GVW9ZQVJlS1hHeTU5eGU1U1FPazJhSjhvKwoyL1l6NVk4R2NOM3pGRTZWaUl2a0duSGhNbEFoWGFYL2JvME05UjYyczAvNnErK1Q3dXdORnVPZzhBPT0KLS0tLS1FTkQgUFVCTElDIEtFWS0tLS0t
    signedIdentity:
      matchPolicy: RemapIdentity
      remapIdentity:
        prefix: example.com
        signedPrefix: mirror.com
  1. Check the upgrade status has Upgradeable=False
$ oc adm upgrade
Cluster version is 4.20.0-0-2025-11-22-160441-test-ci-ln-ftiiw0t-latest

Upgradeable=False

  Reason: ConflictingClusterImagePolicy
  Message: Cluster operator machine-config should not be upgraded between minor versions: ClusterImagePolicy resource named 'openshift' conflicts with the cluster default ClusterImagePolicy object and blocks upgrades. Please delete the 'openshift' ClusterImagePolicy resource and reapply it with a different name if needed

warning: Cannot display available updates:
  Reason: NoChannel
  Message: The update channel has not been configured.

$ oc describe co
Name:         machine-config
Namespace:    
Labels:       <none>
Annotations:  exclude.release.openshift.io/internal-openshift-hosted: true
              include.release.openshift.io/self-managed-high-availability: true
              include.release.openshift.io/single-node-developer: true
API Version:  config.openshift.io/v1
Kind:         ClusterOperator
Metadata:
  Creation Timestamp:  2025-11-22T16:21:48Z
  Generation:          1
  Owner References:
    API Version:     config.openshift.io/v1
    Controller:      true
    Kind:            ClusterVersion
    Name:            version
    UID:             da869125-4279-4cc6-8bf7-41409955328f
  Resource Version:  58798
  UID:               5c1d1c42-856a-44fe-aca0-a1d555d5bfb1
Spec:
Status:
  Conditions:
    Last Transition Time:  2025-11-22T16:31:50Z
    Message:               Cluster version is 4.20.0-0-2025-11-22-160441-test-ci-ln-ftiiw0t-latest
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2025-11-22T16:31:44Z
    Status:                False
    Type:                  Degraded
    Last Transition Time:  2025-11-22T16:31:44Z
    Message:               Cluster has deployed [{operator 4.20.0-0-2025-11-22-160441-test-ci-ln-ftiiw0t-latest} {operator-image registry.build08.ci.openshift.org/ci-ln-ftiiw0t/stable@sha256:bbffb4ebd658ca0e53b1ca7b85ff1a9f748f3f11d1c4f6d129c7906a171d6269}]
    Reason:                AsExpected
    Status:                True
    Type:                  Available
    Last Transition Time:  2025-11-22T18:33:35Z
    Message:               ClusterImagePolicy resource named 'openshift' conflicts with the cluster default ClusterImagePolicy object and blocks upgrades. Please delete the 'openshift' ClusterImagePolicy resource and reapply it with a different name if needed
    Reason:                ConflictingClusterImagePolicy
    Status:                False
    Type:                  Upgradeable
    Last Transition Time:  2025-11-22T16:31:44Z
    Reason:                AsExpected
    Status:                False
    Type:                  EvaluationConditionsDetected
...

Delete the self created testing clusterimagepolicy, turn on the featuregate, should not have Upgradeable=False guard on openshift internal openshift policy

$ oc get clusterimagepolicy
NAME        AGE
openshift   28m
qiwan@fedora:~$ oc adm upgrade
Cluster version is 4.20.0-0-2025-11-22-160441-test-ci-ln-ftiiw0t-latest

Upgradeable=False

  Reason: FeatureGates_RestrictedFeatureGates_TechPreviewNoUpgrade
  Message: Cluster operator config-operator should not be upgraded between minor versions: FeatureGatesUpgradeable: "TechPreviewNoUpgrade" does not allow updates

warning: Cannot display available updates:
  Reason: NoChannel
  Message: The update channel has not been configured.

- Description for the changelog

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 13, 2025
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 13, 2025
@QiWang19 QiWang19 changed the title block upgrades for conflict non-default ClusterImagePolicy resources OCPBUGS-64822: block upgrades for conflict non-default ClusterImagePolicy resources Nov 13, 2025
@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Nov 13, 2025
@openshift-ci-robot
Copy link
Contributor

@QiWang19: This pull request references Jira Issue OCPBUGS-64822, which is invalid:

  • expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

- What I did

- How to verify it

- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@QiWang19
Copy link
Member Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Nov 13, 2025
@openshift-ci-robot
Copy link
Contributor

@QiWang19: This pull request references Jira Issue OCPBUGS-64822, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-64823 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-64823 targets the "4.21.0" version, which is one of the valid target versions: 4.21.0
  • bug has dependents

Requesting review from QA contact:
/cc @asahay19

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested a review from asahay19 November 13, 2025 15:42
@QiWang19
Copy link
Member Author

/test e2e-aws-ovn-techpreview

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 13, 2025

@QiWang19: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test e2e-aws-ovn
/test e2e-aws-ovn-upgrade
/test e2e-gcp-op-1of2
/test e2e-gcp-op-2of2
/test e2e-gcp-op-single-node
/test e2e-hypershift
/test images
/test okd-scos-images
/test periodics-images
/test unit
/test verify
/test verify-deps

The following commands are available to trigger optional jobs:

/test bootstrap-unit
/test e2e-agent-compact-ipv4
/test e2e-aws-disruptive
/test e2e-aws-mco-disruptive
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-fips-op
/test e2e-aws-ovn-ocb-techpreview
/test e2e-aws-ovn-serial-ipsec
/test e2e-aws-ovn-upgrade-ipsec
/test e2e-aws-ovn-upgrade-ocb-techpreview
/test e2e-aws-ovn-upgrade-out-of-change
/test e2e-aws-ovn-windows
/test e2e-aws-ovn-workers-rhel8
/test e2e-aws-proxy
/test e2e-aws-serial
/test e2e-aws-single-node
/test e2e-aws-upgrade-single-node
/test e2e-aws-workers-rhel8
/test e2e-azure
/test e2e-azure-ovn-upgrade
/test e2e-azure-ovn-upgrade-out-of-change
/test e2e-azure-upgrade
/test e2e-gcp-mco-disruptive
/test e2e-gcp-op
/test e2e-gcp-op-ocl
/test e2e-gcp-op-techpreview
/test e2e-gcp-ovn-rt-upgrade
/test e2e-gcp-rt
/test e2e-gcp-rt-op
/test e2e-gcp-single-node
/test e2e-gcp-upgrade
/test e2e-hypershift-techpreview
/test e2e-metal-assisted
/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-ipv6
/test e2e-metal-ovn-two-node-arbiter
/test e2e-metal-ovn-two-node-fencing
/test e2e-openstack
/test e2e-openstack-dualstack
/test e2e-openstack-externallb
/test e2e-openstack-hypershift
/test e2e-openstack-parallel
/test e2e-openstack-singlestackv6
/test e2e-ovirt
/test e2e-ovirt-upgrade
/test e2e-ovn-step-registry
/test e2e-vsphere
/test e2e-vsphere-ovn-upi
/test e2e-vsphere-ovn-upi-zones
/test e2e-vsphere-ovn-zones
/test e2e-vsphere-upgrade
/test okd-scos-e2e-aws-ovn
/test security

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-machine-config-operator-release-4.20-bootstrap-unit
pull-ci-openshift-machine-config-operator-release-4.20-e2e-aws-ovn
pull-ci-openshift-machine-config-operator-release-4.20-e2e-aws-ovn-upgrade
pull-ci-openshift-machine-config-operator-release-4.20-e2e-gcp-op-1of2
pull-ci-openshift-machine-config-operator-release-4.20-e2e-gcp-op-2of2
pull-ci-openshift-machine-config-operator-release-4.20-e2e-gcp-op-single-node
pull-ci-openshift-machine-config-operator-release-4.20-e2e-hypershift
pull-ci-openshift-machine-config-operator-release-4.20-images
pull-ci-openshift-machine-config-operator-release-4.20-okd-scos-e2e-aws-ovn
pull-ci-openshift-machine-config-operator-release-4.20-okd-scos-images
pull-ci-openshift-machine-config-operator-release-4.20-periodics-images
pull-ci-openshift-machine-config-operator-release-4.20-security
pull-ci-openshift-machine-config-operator-release-4.20-unit
pull-ci-openshift-machine-config-operator-release-4.20-verify
pull-ci-openshift-machine-config-operator-release-4.20-verify-deps

In response to this:

/test e2e-aws-ovn-techpreview

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@QiWang19
Copy link
Member Author

/payload-job periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 13, 2025

@QiWang19: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/cc9c00e0-c0a8-11f0-8b11-4184cda435c8-0

@QiWang19 QiWang19 force-pushed the cip-guard-upgrade branch 3 times, most recently from 0ca20b7 to 6203102 Compare November 13, 2025 17:47
@QiWang19
Copy link
Member Author

/payload-job periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 13, 2025

@QiWang19: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/f6a9e450-c0b8-11f0-83d8-cdc5a82ee9c5-0

@QiWang19
Copy link
Member Author

/payload-job periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 17, 2025

@QiWang19: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/c2505fa0-c3f0-11f0-9f37-7713c0102c8d-0

@wking
Copy link
Member

wking commented Nov 17, 2025

Build cluster hiccup:

Trying to pull image-registry.openshift-image-registry.svc:5000/ci-op-1iidzfil/pipeline@sha256:d4f2629dd1eb1700a53020a4b65ece824604f2b5b43e19f65048dc4911f96540...
error: error creating buildah builder: initializing source docker://image-registry.openshift-image-registry.svc:5000/ci-op-1iidzfil/pipeline@sha256:d4f2629dd1eb1700a53020a4b65ece824604f2b5b43e19f65048dc4911f96540: pinging container registry image-registry.openshift-image-registry.svc:5000: Get "https://image-registry.openshift-image-registry.svc:5000/v2/": dial tcp: lookup image-registry.openshift-image-registry.svc on 172.30.0.10:53: no such host

/retest-required

@QiWang19
Copy link
Member Author

/payload-job periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 18, 2025

@QiWang19: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/2c2720c0-c48d-11f0-8ace-4338ea8bfe20-0

@QiWang19
Copy link
Member Author

/payload-job periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 19, 2025

@QiWang19: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/1eeb5b10-c57d-11f0-951d-dbe87bf711a7-0

Signed-off-by: Qi Wang <qiwan@redhat.com>
@QiWang19
Copy link
Member Author

/payload-job periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 21, 2025

@QiWang19: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/2efab780-c71c-11f0-9899-7c55182f5127-0

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 22, 2025

@QiWang19: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/okd-scos-e2e-aws-ovn 6203102 link false /test okd-scos-e2e-aws-ovn
ci/prow/e2e-aws-ovn-upgrade aa13e2d link true /test e2e-aws-ovn-upgrade
ci/prow/bootstrap-unit aa13e2d link false /test bootstrap-unit

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@QiWang19
Copy link
Member Author

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Nov 22, 2025
@openshift-ci-robot
Copy link
Contributor

@QiWang19: This PR has been marked as verified by payloadjob https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-machine-config-operator-5414-ci-4.20-e2e-aws-ovn-techpreview/1991973383591432192.

In response to this:

/verified by payloadjob https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-machine-config-operator-5414-ci-4.20-e2e-aws-ovn-techpreview/1991973383591432192

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@QiWang19
Copy link
Member Author

tests passed, we can use SigstoreImageVerificationPKI as an featureset indicator: if it's disabled, the cluster is on Default feature set.

@QiWang19
Copy link
Member Author

QiWang19 commented Nov 25, 2025

@wking
Copy link
Member

wking commented Nov 26, 2025

I dunno what's going on with bootstrap-unit:

W1121 21:10:14.542506   23204 reconcile.go:68] Failed to fetch MachineConfiguration (will retry): machineconfiguration.operator.openshift.io "cluster" not found
E1121 21:10:43.896306   23204 reflector.go:200] "Failed to watch" err="failed to list *v1.MachineConfiguration: the server could not find the requested resource (get machineconfigurations.operator.openshift.io)" logger="UnhandledError" reflector="github.com/openshift/client-go/operator/informers/externalversions/factory.go:125" type="*v1.MachineConfiguration"
E1121 21:10:48.076566   23204 reflector.go:200] "Failed to watch" err="failed to list *v1.ClusterImagePolicy: the server could not find the requested resource (get clusterimagepolicies.config.openshift.io)" logger="UnhandledError" reflector="github.com/openshift/client-go/config/informers/externalversions/factory.go:125" type="*v1.ClusterImagePolicy"
W1121 21:10:54.542898   23204 reconcile.go:68] Failed to fetch MachineConfiguration (will retry): machineconfiguration.operator.openshift.io "cluster" not found
W1121 21:10:54.542926   23204 reconcile.go:68] Failed to fetch MachineConfiguration (will retry): machineconfiguration.operator.openshift.io "cluster" not found
E1121 21:10:54.542945   23204 render_controller.go:475] Error syncing Generated MCFG: could not generate rendered MachineConfig: unable to fetch the MachineConfigurations object for MC validation: timed out waiting for the condition
E1121 21:10:54.542957   23204 render_controller.go:475] Error syncing Generated MCFG: could not generate rendered MachineConfig: unable to fetch the MachineConfigurations object for MC validation: timed out waiting for the condition

But, the job as a whole seems pretty dead, so maybe unrelated to this change?

Copy link
Member

@wking wking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing with a Cluster Bot launch 4.20,openshift/machine-config-operator#5414 aws cluster (logs):

$ curl -s https://raw.githubusercontent.com/openshift/cluster-update-keys/b3cae8f22b51d9062d0ceb6ac9cf2f7651b4ce8f/manifests.rhel/0000_90_openshift-cluster-image-policy.yaml >policy.yaml
$ oc apply -f policy.yaml
$ oc get -o json clusterimagepolicy openshift | jq -r .metadata.creationTimestamp
2025-11-26T23:36:42Z
$ oc get -o json clusteroperator machine-config | jq '.status.conditions[] | select(.type == "Upgradeable")'
{
  "lastTransitionTime": "2025-11-26T23:36:44Z",
  "message": "ClusterImagePolicy resource named 'openshift' conflicts with the cluster default ClusterImagePolicy object and blocks upgrades. Please delete the 'openshift' ClusterImagePolicy resource and reapply it with a different name if needed",
  "reason": "ConflictingClusterImagePolicy",
  "status": "False",
  "type": "Upgradeable"
}

So looks good to me (and only took 2s to notice :) ). Also picks up deletion:

$ oc delete clusterimagepolicy openshift
$ oc get -o json clusteroperator machine-config | jq '.status.conditions[] | select(.type == "Upgradeable")'
{
  "lastTransitionTime": "2025-11-26T23:40:06Z",
  "reason": "AsExpected",
  "status": "True",
  "type": "Upgradeable"
}

Thanks!

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 26, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 26, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: QiWang19, wking
Once this PR has been reviewed and has the lgtm label, please assign dkhater-redhat for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants